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PREFACE 


This  research  was  accomplished  as  part  of  the  “Space  Logistics  Front-End 
Analysis”  task  order  under  the  Technology  Readiness  and  Sustainment  (TRS) 
contract  (F33615-99-D-6001).  The  period  of  performance  for  this  task  order 
spanned  from  April  2000  through  January  2002. 

The  authors  would  like  to  thank  the  following  individuals  for  their 
enthusiasm  and  contributions  to  this  effort:  John  Cox,  Gracie  Wantland,  Mike 
Newman,  Sue  Glass,  and  Captain  Janet  Haug  from  SMC/AXL;  Dale  McKinzie 
and  Lt  Col  Paul  Scholte  from  AFSPC/LGX;  and  Captain  Tim  Fromm  from  SMC 
DET  1 1  (DAG). 
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Executive  Summary 


This  research  was  accomplished  as  part  of  the  “Space  Logistics  Front-End 
Analysis”  task  order  under  the  Technology  Readiness  and  Sustainment  (TRS) 
contract  (F3361 5-99-D-6001).  The  period  of  performance  for  this  task  order  spanned 
from  3  April  2000  through  31  January  2002. 

The  purpose  of  this  task  was  to  develop  the  framework  for  an  automated 
Logistics  Decision  Support  Tool  (LOST)  to  help  logistics  managers  and  stakeholders 
conduct  timely,  accurate,  and  defendable  logistics  supportability  analyses  for  space 
systems.  The  primary  goals  of  the  LOST  task  included:  1)  identifying  LOST  core 
functional  requirements  2)  performing  a  survey  of  current  logistics  analysis  tools 
related  to  LOST  3)  identifying  potential  technologies  for  LOST,  4)  specifying 
alternative  design  concepts  for  LOST  5)  estimating  a  return  on  investment  for  the 
proposed  LDST  design  concept,  and  6)  developing  a  high  level,  conceptual 
demonstration  to  highlight  the  core  functions  of  the  proposed  LDST  design  concept. 

During  this  task,  meetings  and  discussions  were  conducted  with  SMC  and  HQ 
AFSPC  personnel  to  help  formulate  the  problem  statement,  scope,  and  key 
requirements  for  LDST.  These  interviews  were  supplemented  by  research  of  current 
DoD  and  AF  acquisition  policies  and  guidance  related  to  the  LDST  problem  domain. 
Based  on  these  efforts,  the  following  requirements  were  defined  and  used  as  a 
baseline  for  development  of  the  LDST  concepts  presented  in  this  report: 

■  The  LDST  should  assist  logistics  managers  in  developing  and  evaluating 
logistics  support  concepts  for  space  systems  segments.  A  logistics  support  concept, 
as  defined  for  LDST,  applies  to  a  specific  ILS  element  area  (e.g.  Technical  Data)  and 
includes:  1)  a  recommendation  on  the  specific  products  and  services  required  to 
support  a  space  system  segment,  and  2)  a  source  (organic,  contractor,  or  mix) 
recommendation  for  acquiring  support  products  and/or  performing  support  services 
(e.g.  in-house  or  contractor  training). 

■  The  LDST  should  assist  logistics  managers  (particularly  novice  managers) 
in  developing  logistics  support  concepts  using  a  structured  decision  process  that 
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allows  for  a  rigorous  analysis  of  the  key  decision  factors,  tasks,  and  criteria  (e.g. 
system  requirements,  program  and  technical  risks,  and  other  pertinent  factors)  that 
impact  logistics  product/service,  and  source  recommendations. 

Based  on  these  initial  requirements,  a  follow-on  literature  search  and  survey  of 
applicable  software  technologies  and  tools  was  performed  to  identify  those  that  could 
address  one  or  more  of  the  LOST  requirements.  Most  of  the  technologies  applicable 
to  the  LOST  fall  under  the  umbrella  of  artificial  intelligence,  particularly  knowledge- 
based  or  expert  systems.  In  addition  to  a  review  of  technologies,  an  in-depth  review 
of  current  acquisition  support  tools  was  also  conducted  to  determine  which  existing 
systems  or  tools  contained  features  applicable  to  the  LOST  requirements. 

This  report  addressed  three  alternative  design  concepts  for  LOST,  including 
an  “LOST  Support  Analysis  Checklist,  “LOST  Support  Analysis  Project  Templates”, 
and  a  “LOST  Support  Concept  Generator”.  Each  of  these  concepts  would  involve 
varying  levels  of  complexity,  as  well  as  time  and  effort,  to  implement  as  a  production 
system  or  application.  The  proposed  concept  for  LOST  that  best  addresses  the 
requirements  outlined  above  is  the  “LOST  Support  Analysis  Project  Templates” 
concept  that  would  leverage  the  knowledge  of  logistics  domain  “experts”,  as  well  as 
expert-based  systems  technologies,  to  provide  a  tool  that  can  assist  logistics 
managers  (particularly  novice  managers)  in  identifying  and  addressing  the  specific 
tasks  and  decision  factors  they  need  to  consider  as  part  of  a  supportability  analysis, 
based  on  program  and  system  requirements  for  their  respective  programs.  Logistics 
domain  experts  would  create  and  maintain  the  business  logic  (“rules”)  defining  and 
applying  logistics  support  analysis  task  and  decision  factor  templates  to  a  program 
(a.k.a.  LOST  project)  based  on  the  specific  requirements  of  the  program.  A  Return  on 
Investment  (ROI)  analysis  was  accomplished  to  determine  the  potential  costs, 
benefits,  and  payback  period  for  the  LOST  based  on  this  concept.  Based  on  cost 
estimates  for  the  initial  development  and  annual  sustainment  of  the  LOST,  as  well  as 
expected  savings  in  logistics  training  and  support  analysis  time,  we  estimated  a  three 
to  four-year  payback  period  (“breakeven  point”)  to  recoup  the  initial  investment  in  the 
design,  development,  and  demonstration  of  the  LOST  application. 
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1.  Introduction 

1. 1  Purpose 

This  report  documents  the  results  of  research  activities  associated  with  the 
development  of  a  framework  for  the  Logistics  Decision  Support  Tool  (LOST).  The 
specific  research  activities  conducted  as  part  of  this  effort  included  1)  identifying 
deficiencies  and  shortfalls  associated  with  current  processes  used  to  develop 
logistics  support  concepts  for  space  systems;  2)  specifying  LOST  functional 
requirements  to  help  address  these  deficiencies  and  shortfalls;  3)  performing  a 
survey  of  current  logistics  support  tools  and  software  technologies  that  can  be 
leveraged  on  to  address  LOST  requirements;  4)  estimating  a  return  on  investment  for 
LOST;  and  5)  development  of  design  concepts  to  demonstrate  the  core  functions  of 
the  LOST. 

1.2  Background 

Previous  research  and  discussions  with  logistics  personnel  at  HQ  AFSPC  and 
SMC  personnel  have  indicated  the  need  for  analysis  tools  that  can  help  logistics 
managers  determine  and  document  the  optimal  product  support  strategy  for  space 
system  acquisitions.  The  need  for  such  tools  is  reinforced  to  some  extent  by  recent 
guidance  provided  in  API  63-107,  Integrated  Product  Support  Planning  and 
Management  (Draft),  which  stresses  the  initial  development  and  continuous 
assessment  of  product  support  strategies  throughout  the  life  cycle  of  a  weapon 
system.  According  to  the  API  63-107,  a  key  part  of  the  process  for  developing  a 
product  support  strategy  is  a  “deliberate  evaluation  of  proposed  concepts  and 
practices  against  legislative,  regulatory,  and  other  applicable  decision  criteria”.  The 
quest  to  define  and  develop  a  framework  for  a  decision  support  tool  to  support  such 
an  evaluation  was  the  departure  point  for  the  LOST  research  effort  documented  in 
this  report. 
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1.3  Introduction 


The  principal  objectives  of  Acquisition  Logistics  are  to  ensure  that  support 
considerations  are  an  integral  part  of  the  system’s  engineering  process,  that  the 
system  can  be  cost-effectively  supported  throughout  its  life-cycle,  and  that  the 
support  resources  (products  and  services)  required  to  support  the  system  are 
identified,  developed,  and  acquired.  To  this  end,  numerous  analysis  and  decision 
support  tools,  some  redundant  in  terms  of  functionality,  have  been  developed  to 
support  program  offices,  particularly  logistics  managers\  and  acquire  support  for 
weapon  systems.  For  instance,  there  are  several  DoD,  joint,  and  component 
developed  cost  models  in  existence  for  deriving  weapon  system  life  cycle  cost 
estimates  (e.g.  LCCA),  conducting  network  level  repair  analysis  (e.g.  NRLA)  as  well 
as  numerous  logistics  models  and  analysis  tools  developed  by  specific  programs  to 
support  their  own  unique  tasks  and  program  requirements.  While  these  models  and 
tools  can  be  very  helpful  to  a  logistics  manager  in  “automating”  the  specific  analyses 
and  documentation  requirements  levied  on  program  offices  to  support  acquisition  and 
milestone  decisions,  none  of  these  models  or  tools  directly  support  the  development 
of  alternative  logistics  support  concepts  that  ultimately  relate  program  requirements 
to  decisions  regarding  the  specific  products  and  services  needed  to  support  the 
system  in  the  most  cost-effective  manner  possible,  over  its  entire  life  cycle. 

In  addition  to  these  tools,  there  is  a  myriad  of  DoD,  USAF,  MAJCOM 
directives,  policies,  and  other  instructional  references  available  to  help  logistics 
managers  in  acquiring  and  managing  the  logistics  support  products  and  services 
required  for  a  weapon  system  program.  For  instance,  the  procedures  for  developing 
logistics  inputs  for  an  RFP,  including  defining  and  tailoring  contract  deliverables,  are 
fairly  well  documented  in  references  like  MIL-  HDBK  502,  AFI  10-602,  etc.  However, 
the  majority  guidance  documented  in  these  resources  is  general  in  nature  and 
primarily  speak  to  “what”  a  logistics  manager  needs  to  address  or  consider  in 
acquiring  the  products  and  services  that  comprise  a  support  concept. 


^  “Logistics  Managers”,  as  referenced  in  this  report,  includes  personnel  responsible  for  managing  and 
acquiring  specific  logistics  products  and  services  for  a  weapon  system  -  e.g.  technical  data. 
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What  none  of  these  tools  or  references  provides  is  a  clear  and  concise 
approach  to  help  logistics  managers  perform  a  structured,  traceable,  and  integrated 
supportability  analysis  that  effectively  contributes  to  the  development  of  an  optimal 
product  or  system  support  strategy  as  directed  by  AFI  63-107.  Currently,  the  process 
for  conducting  a  supportability  analysis  has  relied  primarily  on  the  expertise  (through 
limited  training  and  on-the-job  experience)  of  logistics  managers  supporting  a 
program  office,  some  of  who  may  be  working  on  their  first  acquisition  program. 
Hence,  a  majority  of  these  logistics  managers  are  more  generalists,  than  specialists 
or  “experts”  in  any  particular  logistics  domain  (e.g.  Technical  Data).  The  pool  of 
acquisition  logistics  domain  “experts”  is  dwindling,  and  this  problem  will  probably 
become  more  prevalent  in  the  future  based  on  estimates  that  almost  50%  of  the  Air 
Force’s  acquisition  and  sustainment  workforce  become  eligible  to  retire  by  2005. 

To  address  this  projected  loss  of  acquisition  logistics  experience,  as  well  as 
the  fact  that  there  is  no  specific  AF  career  field  for  acquisition  logistics,  a  need  seems 
to  exist  for  a  more  “expert-based”  decision  support  tool  or  application  that  can  help 
logistics  managers,  particularly  novice  managers,  conduct  a  more  structured, 
thorough,  traceable  supportability  analysis  -  one  that  is  tailored  to  the  specific 
requirements  of  their  respective  programs.  This  was  the  premise  of  the  LOST 
research  effort. 

1.4  Scope  of  LOST  Research 

Due  to  the  scope  of  the  unique  requirements  of  space  system  acquisition 
programs,  as  well  as  project  time  and  resource  constraints,  the  research  conducted 
as  part  of  this  task  focused  exclusively  on  defining  a  framework  for  a  decision  support 
tool  that  can  effectively  support  Air  Force  Space  Command  (AFSPC)  logistics 
managers  in  conducting  a  supportability  analysis  for  their  respective  programs  as 
required  by  AFI  63-107.  More  specifically,  the  LDST  framework  discussed  in  this 
report  is  intended  to  help  logistics  managers  in  Air  Force  Space  Systems  program 
offices; 

■  Conduct  a  structured,  logistics  supportability  analysis  that  is  tailored  to  their 
specific  program  and  system  requirements. 
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■  Document  rationale  for  addressing  key  logistics  support  analysis  tasks  and 
decision  factors  (legislative,  DoD,  AF,  AFSPC,  etc.)  required  by  AFI  63-107  that  have 
a  direct  impact  on  system  supportability  and  life  cycle  costs. 

1.5  Study  Participants  and  Meetings 

Appendix  1  identifies  the  primary  participants  in  the  LOST  framework  definition 
and  conceptual  design  effort.  A  program  review  was  held  at  SMC,  Los  Angeles  AFB, 
CA  in  December  2000  with  these  personnel  to  present  an  overview  of  the  LDST 
research  effort,  technical  approach,  etc.  In  addition,  separate  interviews  were 
conducted  with  key  SMC/AXL  personnel  to  become  more  familiar  with  the  type  of 
logistics  support  they  provide  to  SMC  program  offices,  as  well  as  to  gain  additional 
insight  into  the  types  of  documentation  and  other  resources  they  used  to  perform 
their  jobs.  A  final  review  of  the  LDST  research  effort,  as  well  as  a  conceptual 
demonstration  of  the  proposed  LDST  framework  was  presented  at  AFRL/HESS, 
Wright-Patterson  AFB,  OH  in  August  2001. 

2.0  LDST  Requirements  Definition 

2. 1  Problem  Statement  Formulation 

The  first  step  in  defining  the  requirements  for  LDST  was  to  formulate  a  clear, 
concise  problem  statement  that  could  be  used  as  a  departure  point  for  developing 
alternative  solutions,  identifying  enabling  technologies,  etc.  The  LDST  problem 
statement,  articulated  to  some  extent  as  part  of  the  scope  specified  in  Section  1.3, 
was  based  on  a  review  of  a  previous  LDST  development  effort,  as  well  as 
discussions  with  SMC/AXL  and  program  personnel.  The  results  of  this  analysis 
resulted  in  the  formulation  of  the  following  problem  statement  for  LDST : 

“Space  Systems  program  offices  cannot  adequately  justify  and  defend 
logistics  support  decisions  made  during  the  acquisition  process  to  show  that  the 
decisions  made  provide  for  the  delivery  of  a  supportable,  sustainable  system  which 
achieves  the  performance,  cost,  and  schedule  goals  of  the  program” 
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The  interviews  conducted  with  SMC  personnel  highlighted  some  contributing 
factors  to  this  problem  including; 

■  No  specific  AF  career  field  for  “Acquisition  Logistics”  personnel.  Therefore, 
program  office  acquisition  logistics  responsibilities  are  typically  performed  by  logistics 
managers  that  are  more  “generalists”  than  “specialists”  per  se. 

■  Personnel  new  to  the  Air  Force  (e.g.  directly  out  of  Officer  Training  School), 
and  personnel  from  other  USAF  non-acquisition  career  fields,  are  assigned  to  a 
program  office  as  logistics  managers,  responsible  for  managing  the  acquisition  of 
multi-million  dollar  system  support  resources  (e.g.  technical  data). 

■  No  consistent  process  or  framework  for  conducting  a  supportability 
assessment. 

■  No  decision  support  tools  or  applications  to  support  the  development  of 
product  support  concepts. 

The  problem  statement  and  contributing  factors  presented  above  were  used  to 
guide  subsequent  research  activities  for  development  of  an  LOST  framework.  The 
problem  statement  was  included  as  part  of  an  LOST  questionnaire  (see  Appendix  2) 
and  sent  to  a  select  group  of  SMC  logistics  personnel  working  in  various  SMC 
program  offices  to  support  further  development  and  refinement  of  the  LOST 
framework.  Formal  responses  to  the  questionnaire  were  extremely  limited. 
Therefore,  a  follow-up  telecon  discussion  was  conducted  with  some  of  these 
personnel  to  reiterate  the  intent  of  the  questionnaire,  and  solicit  feedback  on  specific 
questions  related  to  the  LOST  problem  statement,  inputs,  and  outputs.  The 
information  acquired  from  this  discussion  was  used  to  support  the  development  of  the 
LOST  framework. 

2.2  Identifying  LOST  Objectives,  Key  Capabilities,  and  Outputs 

During  the  IPR  meeting  convened  at  SMC  in  December  2000,  key  personnel 
from  HQ  AFSPC/LGXR,  SMC/AXL,  and  logistics  managers  in  SMC  program  offices, 
participated  in  a  Nominal  Group  Technique  (NGT)  session  (modified  to  accommodate 
the  time  constraints  of  the  meeting)  to  help  scope  the  overall  objectives  and 
requirements  for  the  LDST  effort.  The  NGT  session  provided  an  avenue  for  rapidly 
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gathering  and  ranking  ideas  that  extended  beyond  the  LOST  problem  statement,  to 
focus  more  attention  on  three  key  components  of  the  LOST  requirements  definition 
process.  These  components  included  identifying  and  prioritizing; 

■  The  primary  objectives  of  the  LOST 

■  The  key  output  products  of  the  LOST 

■  The  important  capabilities  or  core  functions  of  the  LOST 

Due  to  the  time  constraints  of  the  NGT  session,  and  availability  of  personnel,  these 
were  the  only  three  topic  areas  evaluated  during  the  NGT  session. 

The  NGT  session  provided  a  forum  for  participants  to  have  equal  opportunity 
in  contributing  ideas  for  each  of  three  LOST  requirement  topic  areas  identified  above, 
followed  by  short  discussions  to  clarify  and  consolidate  inputs  as  necessary,  and 
finally,  allowing  each  participant  to  rate  the  top  five  ideas  generated  for  each  topic 
area.  The  rating  scheme  used  to  prioritize  the  ideas  submitted  by  participants  was 
fairly  simple.  During  the  rating  of  each  of  the  three  topics,  each  participant  was 
asked  to  select  what  they  deemed  to  be  the  top  five  inputs  and  assign  a  numeric 
rating  from  “1”  (least  important)  to  “5”  (most  important).  The  numerical  ratings 
assigned  by  participants  were  tallied  for  each  response  to  derive  a  total  score  for  the 
same.  The  synopsis  provided  below  highlights  the  responses  (rank  ordered  based  on 
points)  for  each  topic  question: 

Question  1 :  “What  are  the  primary  objectives  of  the  LDST?” 

■  Provide  a  common,  traceable,  acceptable,  and  reportable  process  for 
supportability  decisions.  A  process  that  ensures  all  ILS  elements  are  addressed 
using  accepted  policy  and  procedures  (30  pts) 

■  Help  acquisition  logistics  and  program  managers  make  cost-effective, 
support  decisions  (16  pts) 

■  Provide  the  capability  to  perform  what-if  analysis  (1 2  pts) 

■  Support  collaboration  among  stakeholders  (11  pts) 

■  Support  for  inexperienced  logisticians  via  computer-based  training  (10  pts) 
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■  Lessons  learned  database  (6  pts) 

■  Common  -  Open  architecture  easy  to  use  (3  pts) 

■  Provide  status  of  decision  activities  (1  pt) 

The  top  three  inputs  listed  above  represent  67%  of  the  total  points  generated 
for  this  question.  The  first  response  seems  to  clearly  indicate  the  need  for  a  more 
formal,  structured,  and  rigorous  process  for  determining  system  support 
requirements,  which  aligns  with  the  problem  statement  defined  earlier  in  the  report  for 
LOST. 

Question  2:  “What  are  the  most  important  output  products  of  an  LOST?” 

■  Tasks  to  be  performed  in  each  ILS  area  (19  pts) 

■  Organic  and  contractor  requirements  (15  pts) 

■  Support  the  output  of  decision  option  reports  including  associated 
tasks/cost  analysis  (15  pts) 

■  Capture  decisions  at  a  "top  level".  For  example,  will  the  system  be  COTS 
or  developmental?  (13  pts) 

■  Cost  data  for  each  ILS  element  (1 1  pts) 

■  Analysis  summary  (2  pts) 

■  List  of  contract  Deliverables  (2  pts) 

■  Lessons  Learned  (1  pt) 

The  first  three  inputs  represent  63%  of  the  total  points  assigned  to  all 
responses.  It  is  worth  noting  that  the  response,  “tasks  to  be  performed  in  each  ILS 
area”,  is  at  the  very  essence  of  defining  a  structured  process  (guided  by  business 
rules)  for  determining  system  support  requirements,  particularly  for  an  “expert” 
based,  decision  support  system.  This  is  a  key  part  of  developing  a  framework  for  the 
LOST  and  formulating  LOST  design  concepts. 

Question  3:  ‘What  are  the  key  capabilities  or  functions  of  an  LOST?” 
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■  Support  collaboration  (20  pts) 

■  Hyperlink  to  source/reference  documents  (1 0  pts) 

■  Interface  to  existing  cost  models  (8  pts) 

■  Capability  to  update  decision  factors  (8  pts) 

■  Configuration  management  control  (6  pts) 

■  Work  on  partial  data  (5  pts) 

■  Support  MS  office  exchange  for  e-mail  (4  pts) 

■  Document  decisions,  comments,  etc  (4  pts) 

■  Support  a  what-if  analysis  capability  (3  pts) 

■  User  Admin  module  to  control  level  of  access,  read/write  capability,  etc.  (3 
pts) 

■  Track  changes  /  action  items  (2  pts) 

■  Web-based  tool  or  application  (2  pts) 

The  top  three  inputs  listed  above  represent  61%  of  the  total  rating  points 
generated  for  all  inputs  to  this  question.  The  “support  collaboration”  response  was 
not  completely  defined,  but  in  general,  means  the  ability  for  other  stakeholders  in  the 
acquisition  process  (e.g.  HQ  AFSPC)  to  actively  participate  in  the  acquisition  logistics 
process  such  as  determining  and  coordinating  on  support  requirements  that  may 
become  part  of  an  RFP. 

2.3  Discussions  with  SMC  Acquisition  Logistics  Personnel 

As  stated  earlier,  separate  meetings  were  conducted  with  a  select  group  of 
SMC/AXL  personnel  with  experience  in  specific  ILS  areas,  to  include  Maintenance 
Planning,  Technical  Data,  and  Training  and  Training  Systems.  The  purpose  of  these 
meetings  was  to  become  generally  familiar  with  the  type  of  support  these  personnel 
provide  to  program  offices  as  logistics  managers  for  each  of  their  respective  ILS 
areas,  as  well  as  to  gain  additional  insight  on  how  a  decision  support  tool  could  help 
them  be  more  effective  in  their  jobs.  It  was  evident  from  these  discussions  there  was 
no  structured  process,  applied  consistently  across  programs,  for  assessing  system 
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support  requirements.  In  addition,  domain  specific  (e.g.  Technical  Data)  logistics 
expertise  and  knowledge  is  typically  accrued  over  time  on  a  “trial  by  fire”  basis,  and 
the  number  of  resident  or  staff  experts  is  extremely  limited.  Hence,  an  AXL  resident 
“expert”  in  a  logistics  area  such  as  Technical  Data  may  be  supporting  several 
program  offices  at  one  time  (source  selections,  staff  assistance  support  to  novice 
personnel,  etc.).  This  does  not  mean  that  a  decision  support  tool  like  LDST  is  going 
to  be  a  panacea  for  ensuring  program  offices  identify,  document  (particularly  in 
RFPs),  and  defend  support  requirements  more  effectively  and  consistently  than 
current  practices  seem  to  indicate,  but  rather,  that  there  is  considerable  room  for 
improving  the  process  of  assessing  the  logistics  support  requirements  for  space 
systems,  particularly  since  the  pool  of  expertise  available  to  program  offices  in  each 
ILS  area  is  limited. 

2.4  Summary  of  Key  LDST  Functional  Requirements 

Based  on  the  original  problem  statement  developed  for  the  LDST,  as  well  as 
the  information  provided  through  discussions  with  HQ  AFSPC  and  SMC  personnel, 
including  the  results  from  the  NOT  session,  the  following  “top-level”  functional 
requirements  were  formulated  to  support  further  research  and  development  of  the 
LDST  framework  and  design  concepts: 

■  The  LDST  should  provide  an  automated  capability  to  support  a  more 
structured,  traceable  process  for  conducting  logistics  support  analyses  that  helps 
ensure  key  tasks  and  decision  factors  impacting  system  sustainment  are  addressed. 

■  The  LDST  should  provide  more  effective  support  to  novice  or  in¬ 
experienced  logisticians  responsible  for  logistics  support  analyses  activities  in  a 
program  office  (reduce  reliance  on  a  diminishing  pool  of  ILS  domain  “experts”). 

■  The  LDST  should  be  implemented  in  a  web-based  environment  to  promote 
more  effective  collaboration  between  all  program  stakeholders  (Using  Commands, 
HQ  AFSPC,  SMC  program  offices,  etc.) 
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■  The  LOST  should  allow  logistics  managers  to  document  the  results  of  the 
analysis  undertaken  in  each  ILS  area  to  address  key  tasks  and  decision  factors  that 
contribute  to  the  development  of  an  overall  product  support  strategy. 

These  initial  LOST  requirements  were  used  to  guide  the  development  of  a 
framework  (design  concept)  for  a  software  application  that  can  help  keep  logistics 
managers  focused  on  key  logistics  decision  factors  and  program  requirements  that 
could  have  a  significant  impact  on  system  sustainment.  For  instance,  if  an  ORD 
requirement  stipulates  a  preference  to  use  contractor  support  for  depot  maintenance, 
that  may  drive  the  need  to  ensure  that  a  program  office  acquires  depot  level  technical 
manuals,  evaluates  alternatives  for  using  contractor  or  organic  depot  facilities,  etc. 

While  these  two  “top-level”  functional  requirements  do  not  address  all  the 
inputs  generated  from  discussions  and  interviews  with  SMC  and  HQ  AFSPC 
personnel,  as  well  as  the  ideas  generated  during  the  NGT  session,  they  do  focus  on 
addressing  the  objective  of  providing  a  common,  traceable,  acceptable,  and 
reportable  process  for  supportability  decisions  -  a  process  that  ensures  all  ILS 
elements  are  addressed  using  accepted  policy  and  procedures.  This  was  ranked  as 
the  number  one  priority  objective  for  the  LOST.  Other  recommendations  from  the 
NGT  session,  including  the  capability  to  perform  a  cost  assessment  of  support 
concepts,  an  “what-if  analyses”,  etc,  may  become  important  extensions  to  an  LOST, 
but  do  not  directly  address  the  LOST  problem  statement,  and  were  considered 
beyond  the  scope  of  the  current  LOST  research  effort. 

3.0  Survey  of  Logistics  Decision  Support  Tools 

There  are  more  than  100  separate  software  tools  (some  developed  as 
prototypes  only)  that  have  been  developed  over  the  years  to  support  specific  tasks  or 
activities  that  support  the  business  of  acquisition  and  sustainment  logistics.  A 
considerable  amount  of  the  LOST  research  effort  was  devoted  to  a  literature  search 
and  assessment  of  tools,  models,  and  applications  that  possessed  certain  functional 
or  technical  characteristics  applicable  to  the  LOST  requirements  outlined  above  in 
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Section  2.4.  A  synopsis  of  the  more  relevant  tools,  models,  or  applications, 
applicable  to  the  requirements  of  LOST  follows. 

3. 1  Logistics  Planning  and  Requirements  System  (LOG PARS) 

Description.  LOGPARS  is  a  PC-based,  tri-service,  expert  system  for 
assisting  program  managers  in  the  preparation  of  integrated  acquisition  planning 
documentation.  It  is  designed  to  enhance  productivity  and  accuracy  in  acquisition 
planning  and  performance  by  leading  the  user  through  the  process  of  establishing 
the  appropriate  acquisition  strategy  and  developing  tailored  supportability  planning 
and  scheduling  documentation.  LOGPARS  is  approved  by  Headquarters  Army 
Materiel  Command  (HQ  AMC)  for  use  in  program  management  offices  and  is  listed  in 
AMC  Regulation  700-15.  The  sponsor  of  LOGPARS  is  the  USAMC  Logistics  Support 
Activity  (LOGSA).  The  primary  purpose  of  the  tool  is  acquisition  and  supportability 
planning. 

Features.  LOGPARS  is  an  expert  based  system  that  prompts  a  user  through 
interactive  question  and  answer  sessions  to  address  appropriate  issues  necessary  to 
automatically  generate  program  documentation.  Modules  are  available  to  assist  in 
the  preparation  of  the  following  planning  documentation:  Acquisition  Strategy, 
Supportability  Strategy,  Material  Fielding  Plan,  ILS  Statement  of  Work,  Provisioning 
Plan,  Transportability  Report,  Warranty  Clause,  and  Material  Fielding  Plan. 

The  LOGPARS  software  consists  of  an  expert  shell  (referred  to  as 
DOCSHELL),  which  includes  an  inference  engine,  and  an  integrated  knowledge  base 
(KB).  The  decision  rules  embedded  in  a  knowledge  base  to  prompt  users  through  the 
applicable  questions  they  need  to  address  to  produce  respective  program 
documentation.  Using  the  decision  logic  in  LOGPARS,  the  knowledge  base 
determines  the  order  and  applicability  of  questions  and  can  even  recommend 
answers.  When  applicable,  LOGPARS  issues  advisory  statements  during  the 
preparation  of  a  given  document  to  maintain  referential  integrity  within,  and  between 
document  modules  for  a  given  program.  Software  programming  skills  are  not 
required  to  revise  or  expand  modules  that  comprise  the  KB.  The  KB  drives  the 
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LOGPARS  expert  system  and  incorporates  the  experience  of  Integrated  Logistics 
Support  (ILS)  planning  expertise,  lessons  learned,  and  the  current  acquisition 
policies/procedures. 

LOGPARS  runs  in  a  multi-user  environment  and  supports  team  efforts  by 
allowing  other  program  personnel  to  participate  in  the  development  of  program 
documents.  In  addition,  draft  documents  can  be  staffed  electronically  and  reviewers 
are  provided  with  the  capability  to  input  comments  on  a  document. 

Applicability  to  LOST.  The  expert-based  framework  used  to  develop 
LOGPARS  is  similar  to  the  approach  envisioned  for  the  LOST.  Although  the  intent  of 
the  LOST  is  not  to  serve  as  a  document  generator,  it  is  possible  that  the  process 
used  by  LOGSA  to  develop  the  decision  rules  and  logic  in  the  LOGPARS  KB, 
particularly  for  the  development  of  the  “Supportability  Strategy”  document,  and 
possibly  some  of  the  decision  rules  themselves,  may  be  applicable  to  the  LOST 
effort.  It  is  also  possible  that  the  DOCSHELL  component,  and  knowledge 
development  environment  used  to  develop  LOGPARS  decision  rules,  could  be 
obtained  (without  any  cost  to  the  government)  and  used  to  prototype  the 
development  of  LOST  decision  rules. 

3. 2  Post  Fielding  Support  Analysis  (PFSA) 

Description.  PFSA  is  another  LOGSA  product  currently  under  development 
as  part  of  a  tri-service  initiative  to  improve  and  streamline  the  ILS  process.  The 
overall  objective  of  PFSA  is  to  create  an  integrated  analysis  environment  that 
enables  the  services  to  be  both  responsive  and  proactive  in  managing  logistics 
support  and  reducing  O&S  costs.  This  includes  streamlining  the  ILS  process  by 
providing  field  units  direct  access  and  insight  into  the  product  sustainment  process 
(e.g.  processing  of  Engineering  Change  Proposals  -  ECPs),  and  direct  contact  with 
logistics  support  personnel  (e.g.  item  managers).  This  will  allow  personnel  at  the  unit 
level  to  track  the  progress  of  actions  related  to  the  reporting  of  materiel  deficiencies, 
ECPs,  etc.,  as  well  support  continuous,  automated  monitoring  of  critical  metrics.  The 
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PFSA  initiative  will  utilize  existing  models  and  analytical  tools,  as  applicable,  as  well 
as  existing  data  sources. 

Features.  PFSA  is  being  designed  to  run  as  a  PC-based,  client-server 
application  (can  also  be  invoked  through  a  web-interface  to  act  as  a  “thin”  PC  client). 
PFSA  contains  four  distinct  modules  including  1)  a  problem-reporting  module,  2)  an 
information  module,  3)  a  logistics  analysis  module,  and  4)  a  reports  module.  The 

i 

design  plan  for  PFSA  will  incorporate  a  knowledge  base,  and  the  use  of  intelligent 
agents  to  source  data,  monitor  and  report  activity  status  based  on  customized  user 
profiles,  as  well  as  make  specific  analysis  recommendations  (“intelligent  analysis”) 
based  on  the  type  of  reported  problem  or  deficiency. 

I 

i 

Applicability  to  LOST.  The  logistics  analysis  module  in  PFSA  includes  three 
sub-modules  that  could  potentially  be  included  as  part  of  an  LOST  analysis.  These 
modules  include  a  cost-driver  analysis,  level  of  repair  analysis,  and  predictive  trend 
analysis  that  tracks  program  metrics  based  on  input  from  an  ORD.  It  is  possible  that 
the  approach  used  to  develop  an  “intelligent  analysis”,  as  well  as  components  from 
one  or  all  of  these  modules  may  provide  some  utility  in  an  LOST  analysis. 

3.3  Engineering  Support  Tool  (EST) 

Description.  The  Engineering  Support  Tool  was  developed  by  SMC/XR  to 
assist  logisticians  in  the  developmental  planning  process  by  helping  ensure  system 
support  requirements  are  addressed  in  a  “systematic”  manner  early  on  in  a  program. 
It  is  intended  to  help  logistician’s  develop  and  assess  alternative  support  concepts  for 
new  technologies  that  could  potentially  improve  the  performance  and  life-cycle  costs 
of  new  systems.  The  EST  supports  an  assessment  of  pre-defined,  qualitative 
technology  “attributes”,  against  “operational  suitability”  criteria  that  include: 
dependability,  availability,  sustainability,  maintainability,  producibility,  affordability, 
testability,  human  factors,  and  environment. 
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Features.  The  EST  is  implemented  in  the  form  of  an  MS-Excel  spreadsheet 
that  supports  the  inputs  of  technology  attribute  ratings  and  supporting  justification,  as 
well  as  the  calculation  of  an  overall  “operational  suitability”  index  for  each  technology. 

Applicability  to  LOST.  It  is  possible  that  some  of  the  criteria  used  to  further 
define  each  technology  attribute  considered  by  EST  may  be  applicable  to  the 
development  of  decision  rules  and  “facts”  that  would  comprise  an  LOST  knowledge 
base. 

3.4  LOST  Version  1.0  (Prototype) 

Description.  In  1998,  bd  Systems  Inc.  worked  with  SMC  personnel  to 
design  and  develop  a  prototype  version  of  LOST  that  could  be  used  to  assist  the  Air 
Force  Materiel  Command  acquisition  logistics  manager,  and  Air  Force  Space 
Command  in  determining  the  most  appropriate  support  plan  for  each  space  system, 
or  segment,  under  development.  The  prototype  tool  was  intended  to  assist  a  user 
by  providing  a  decision  logic  process  that  can  be  documented  and  supported.  The 
decision  logic  in  the  prototype  LOST  is  encapsulated  in  a  decision  tree  template  that 
steps  the  user  through  a  series  of  program  and  logistics  support  analysis  questions 
and  records  user’s  responses.  The  LDST  logic  tree  supports  a  separate 
assessment  of  two  levels  of  maintenance:  organizational  level  maintenance,  and 
depot  (or  factory  level)  maintenance.  The  decision  tree  analysis  method  is  used  to 
evaluate  a  variety  of  support  alternatives  for  each  level  of  sustaining  maintenance 
and  repair.  Each  path  in  the  decision  tree  process  may  be  documented  for  report 
generation,  as  well  as  for  use  in  comparative  analysis  with  other  decision  paths  as  a 
means  to  substantiate  and  justify  the  recommended  decisions. 

Features.  The  prototype  version  of  the  LDST  developed  under  this  earlier 
effort  is  a  PC  application  that  provides  limited  support  for  the  creation  and  editing  of 
decision  support  templates  used  to  conduct  a  logistics  support  analysis.  The 
prototype  version  produces  two  types  of  output  reports.  The  first  report  is  the 
“Decision  Logic  Report”,  which  shows  an  individual  run  with  the  questions  and 
selected  answers.  This  is  useful  for  documenting  a  decision.  It  may  also  be  used 
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within  a  team  to  allow  others  to  comment  on  the  process.  The  second  report  is  the 
“Comparison  Report”,  which  shows  the  points  at  which  two  runs  diverge.  It  does  not 
show  the  complete  runs.  This  feature  is  useful  in  understanding  which  decision 
influenced  differing  outcomes  of  the  process. 

Applicability  to  LOST.  To  our  knowledge,  development  of  the  LOST 
prototype  version  was  not  extended  beyond  this  initial  effort.  However,  data  and 
information  collected  as  part  of  the  requirements  phase  of  the  prototype  effort, 
particularly  some  of  the  ILS  element  specific  decision  criteria,  and  program  risk 
factors  could  be  incorporated  (as  applicable)  as  part  of  the  LOST  framework 
developed  under  this  research  effort. 

3.5  Performance  SupportabUity  Metric  (PSM) 

Description.  The  PSM  is  a  risk-based  management  tool  developed  and 
delivered  by  bd  Systems  Inc.  in  August  2001  as  a  replacement  to  the  previous  AFMC 
Command  Support  Metric  (CSM)  application.  The  CSM  was  intended  to  help 
program  managers  and  senior  management  personnel  assess  the  “health”  of 
supportability  for  a  specific  program.  The  CSM  focused  almost  exclusively  on 
tracking  the  planning  and  accomplishment  of  key  program  events  and  activities 
related  to  logistics  (e.g.  Maintenance  Concept  Approved).  The  primary  shortfall  of 
CSM  that  is  addressed  in  PSM  is  program  risks.  PSM  attempts  to  assess  the  overall 
impact  and  potential  risks  to  a  program  if  an  event  is  not  completed  on  time,  or 
delayed  to  some  future  time  period. 

Features.  The  PSM  tool  provides  for  risk-based  “expert”  management  of 
program  supportability  activities  and  events,  and  allows  logisticians  to  measure  ILS 
program  risks.  It  also  facilitates  risk  mitigation  planning,  and  allows  logistics 
managers  to  predict,  analyze,  manage  status,  and  report  on  the  health  of  their 
logistics  program. 

Applicability  to  LOST.  The  PSM  is  focused  on  tracking  and  assessing  risks 
associated  with  key  program  events  that  have  a  direct  impact  on  logistics.  It  does  not 
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however,  address  the  matter  of  helping  a  logistics  manager  identify,  organize,  and 
document  the  specific  logistics  support  analysis  tasks  and  decision  factors  that 
should  be  addressed  for  a  particular  new  system  acquisition  or  modification.  In  this 
sense,  the  LOST  could  feed  PSM  inputs  on  the  status  and  actions  taken  by  logistics 
managers  to  address  key  decision  factors  and  activities  throughout  a  program. 

4.0  Potential  Technologies  Applicable  to  LOST 

Based  on  the  on  the  baseline  requirements  specified  in  Section  2  of  this  report 
for  development  of  the  LOST  framework,  a  survey  and  review  of  applicable  software 
technologies  was  conducted.  This  survey  focused  on  software  technologies  in  the 
artificial  intelligence  (Al)  domain,  primarily  knowledge-based  or  expert  system 
software  tools  that  have  been  developed  and  applied  in  various  domains  to  help 
humans  solve  complex  problems,  like  those  we  expect  to  be  confronted  with  in 
development  of  the  LOST.  Before  discussing  some  of  the  variant  approaches  for 
expert  system  development,  a  brief  discussion  about  expert  systems  is  probably  in 
order. 

Expert  systems  are  computer  programs  that  simulate  the  reasoning  process 
and  knowledge  of  human  “experts”  to  solve  specific  problems  (Lee,  1988,  Turban, 
1990).  One  of  the  distinct  features  of  expert  systems  is  that  they  support  the 
modeling  of  information  at  higher  levels  of  abstraction,  and  are  developed  in  a 
manner  that  closely  resembles  human  logic.  The  goal  of  most  expert  systems  is  to 
improve  organizational  efficiency  and  effectiveness  (Ashmore,  1989).  For  instance, 
in  the  case  of  LOST,  the  goal  for  developing  an  expert  system  would  be  to  emulate 
the  process  that  logistics  “experts”  follow  to  make  decisions  about  the  products  and 
services  needed  to  support  a  system,  so  that  in-turn,  novice  logistics  managers  could 
perform  the  same  task  more  efficiently  and  effectively,  while  at  the  same  time 
improving  their  problem-solving  skills.  Since  expert  systems  are  intended  to  simulate 
a  human  expert,  they  also  have  the  potential  to  reduce  the  time  devoted  to  solving 
complex  problems,  while  looking  at  it  the  problem  from  a  variety  of  perspectives. 
Some  of  the  more  common  applications  of  expert  systems  include:  strategic  goal 
setting,  planning,  design,  scheduling,  monitoring,  and  diagnostics. 
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The  key  components  of  simplified  expert  system  architecture  are  depicted  in 
Figure  1.  These  components  include  a  user  interface,  knowledge  base  (KB),  and 
inference  engine  -  which  together  comprise  what  can  be  referred  to  as  a  KB 
development  environment.  Hence,  Figure  1  is  intended  to  convey  that  these  two 
components  are  closely  integrated  and  usually  comprise  the  software  required  to 
develop  expert  systems.  It  is  also  worth  noting  from  Figure  1  that  users  do  not 
interface  directly  with  the  inference  engine  of  an  expert  system,  but  rather  through  a 


graphical  user  interface. 


Processing  sequence  driven  by  interaction 
between  the  user  and  knowledge  base 


Contains  the  specific  facts  and  heuristics,  as 
well  as  the  rules  to  relate  this  information  to 
user  Inputs.  Returns  recommendations  and 
decisions  from  the  inference  engine 


Automatically  matches  facts  against 
patterns  and  determines  which  rules  are 
applicable  (or  not  applicable) 


Figure  1.  Expert  System  Basic  Architecture  -  Simplified  Representation 

The  KB  is  arguably  the  most  important  component  in  an  expert  system,  and 
probably  the  most  significant  challenge  in  the  design  and  development  of  any  expert 
system.  The  KB  incorporates  the  rules,  expertise,  know-how,  procedures,  policies 
and  regulations  that  support  an  expert  system.  A  majority  of  the  expert  system  KBs 
in  existence  today  is  implemented  using  one  of  three  approaches,  namely  decision 
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trees,  case-based  reasoning,  and  rule-based  expert  systems.  These  approaches  are 
summarized  below. 

4.1  Decision  Trees 

Decision  trees  graphically  portray  a  hierarchical  set  of  rules  that  describe  how 
a  person  might  evaluate  or  classify  an  object  of  interest  based  on  the  answers  to  a 
series  of  questions  (typically  “yes”  or  “no”  questions).  The  answer  provided  to  each 
question  dictates  the  branching  path  taken  through  the  tree  that  ultimately  leads  to  an 
end  result  or  decision.  Because  decision  trees  are  graphical  (visual)  in  nature,  they 
tend  to  have  the  advantage  of  being  easy  to  understand  and  explain,  and  can 
execute  decisions  processes  fairly  rapidly.  However,  one  of  the  primary 
disadvantages  of  using  a  decision  tree  for  knowledge  base  development  is  that  they 
maintain  a  precise  logic  hierarchy  that  does  not  lend  itself  to  being  easily  modified  as 
questions  are  added  or  deleted  from  the  tree.  Since  a  decision  tree  for  LOST  could 
be  very  large  and  complex,  and  highly  likely  that  the  decision  logic  will  change  from 
time  to  time  as  new  knowledge  is  acquired  that  impacts  the  process  of  developing 
logistics  support  concepts  (decision  factors,  risks,  policies,  and  lessons  learned),  it 
does  not  readily  seem  like  the  best  approach  from  a  software  maintenance 
standpoint  for  implementing  an  LOST  KB. 

4.2  Case-Based  Reasoning 

For  the  most  part,  case-based  reasoning  (CBR)  is  synonymous  to  “reasoning 
by  analogy”.  The  goal  of  CBR  is  to  solve  a  current  problem  by  retrieving  solutions  to 
historical  problems,  that  are  similar  in  nature,  and  then  modifying  those  same 
solutions  to  derive  a  solution  that  addresses  the  current  problem.  The  historical 
solutions  selected  from  the  knowledge  base  and  used  to  form  the  solution  to  the 
current  problem  are  typically  selected  based  on  some  weighting  or  probabilistic 
assignment  scheme.  Hence,  CBR  is  based  on  the  previous  experiences  of  human 
experts  that  have  years  of  experience  in  a  particular  job  and  activity.  In  the  case  of 
the  LOST,  where  “expert”  level  experience  in  performing  jobs  like  managing  the 
acquisition  of  system  Technical  Data  is  becoming  a  scarce  commodity,  CBR  could 


prove  to  be  a  very  relevant  technique  for  development  of  a  LOST  knowledge  base. 
This  could  prove  to  be  a  distinct  advantage  for  novice  logistics  managers  who  could 
draw  on  the  knowledge  of  more  experienced  colleagues,  including  the  knowledge  of 
personnel  that  are  no  longer  in  the  organization  to  readily  help  them  solve  their 
problems.  In  addition,  CBR  avoids  some  of  the  maintenance  pitfalls  associated  with 
decision  trees,  since  new  questions  and  answers  can  be  added  in  incremental 
fashion  to  the  knowledge  base  without  regard  for  the  problem  of  ensuring  the  logical, 
precise  positioning  of  new  questions  encountered  when  using  decision  trees.  Hence, 
for  problem  domains  where  the  decision  logic  is  quite  extensive  and  complex,  CBR 
may  be  a  better  choice  for  KB  development  than  decision  trees.  However,  even 
though  CBR  avoids  some  of  the  pitfalls  encountered  with  the  use  of  decision  trees, 
there  is  still  a  considerable  amount  of  human  effort  and  time  required  to  initially 
develop  an  extensive  “library”  of  solutions  depending  on  the  complexity  of  the 
problem  domain,  and  the  knowledge  must  be  more  rigorously  structured  than  simply 
a  text  format. 

4.3  Rules-based  Programming 

Rules-based  programming  is  probably  one  of  the  most  commonly  used 
techniques  for  developing  KBs  for  expert  systems.  This  programming  technique  uses 
rules  to  represent  heuristics,  or  "rules  of  thumb,"  that  specify  a  specific  set  of  actions 
to  be  performed  based  on  the  state  of  events  in  a  given  situation.  For  example,  if  the 
doorbell  rings,  then  answer  the  door.  Rules  are  expressed  in  an  imperative  or 
declarative  manner  using  a  combination  of  “if-then”  statements  to  form  a  rule.  The  “if 
portion  of  a  rule  is  a  series  of  patterns  which  specify  the  facts  (or  data)  to  determine 
the  applicability  of  a  rule.  The  process  of  matching  facts  to  patterns  is  called  pattern 
matching  and  is  handled  by  the  inference  engine  of  the  expert  system.  The  “then” 
portion  of  a  rule  is  the  set  of  actions  to  be  executed  when  the  conditions  of  the  “if 
portion  of  the  rule  are  satisfied.  The  applicable  actions  pertaining  to  the  rule  are 
executed  when  the  inference  engine  is  instructed  to  begin  execution.  The  inference 
engine  then  selects  another  rule  and  executes  its  actions.  This  process  continues 
until  no  applicable  rules  remain. 
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The  rules-based  approach  is  more  flexible  than  a  decision-tree  approach,  but 
from  a  software  execution  standpoint,  may  require  more  processing  time  than  a 
decision-tree  or  CBR  approach  if  there  are  numerous  rules  in  the  KB,  each  of  which 
would  need  to  be  examined  to  determine  if  the  facts  match  the  selection  criteria 
encompassed  as  part  of  the  “if  portion  of  the  decision  rule. 

4.4  Expert  System  Software  Development  Tools 

There  are  several  commercial  and  government  software  tools  available  to 
support  the  development  of  expert  systems,  including  the  LOGSA  DOCSHELL 
knowledge  development  environment,  discussed  in  Section  3.  A  survey  of  existing 
expert  system  development  tools  is  being  conducted  to  determine  which  tools  could 
potentially  support  the  development  of  LOST.  To  date,  the  following  expert  system 
software  packages  have  been  identified  as  potential  candidates  for  LOST; 

■  Sawion  Business  Manager,  Sawion  Corp. 

This  is  a  rules-based;  expert  system  development  tool  that  provides  the 
capability  to  define  and  capture  step-by-step  procedures  and  information  flows  in  a 
process,  as  well  as  the  business  rules  and  policies  supporting  each  step  in  a 
process.  Expert  systems  developed  with  this  product  can  be  deployed  in  a 
LAN/WAN  or  Internet  environment. 

■  Cyc  Knowledge  Server,  Cycorp  Corp. 

The  “Cyc  Knowledge  Server”  is  a  large,  multi-contextual  knowledge  base  and 
inference  engine  used  to  develop  expert  systems.  It  supports  natural  language 
processing,  and  provides  the  capability  to  query,  browse,  and  edit  information  in  the 
knowledge  base. 

-  CLIPS 

CLIPS  is  an  expert  system  tool,  which  provides  a  complete  environment  for 
the  construction  or  rule-based,  and  object-based  expert  systems.  It  supports  three 


20 


different  programming  paradigms  for  developing  expert  systems,  including  rule- 
based,  object  oriented,  and  procedural  programming  languages  (C,  Pascal,  LISP, 
Ada).  The  software  is  written  in  the  “C”  programming  language  to  support  fast 
execution  and  platform  portability.  The  CLIPS  software  has  supported  numerous 
government  and  commercial  users,  including  NASA,  DoD  service  components, 
universities,  etc. 

■  JESS  (Java  Expert  System  Shell),  Sandia  National  Laboratories 

JESS  is  an  expert  system  shell  written  entirely  in  Java.  Jess  supports  the 
development  of  rule-based  expert  systems,  which  can  be  tightly  coupled  to  code 
written  in  the  powerful,  portable  Java  language.  JESS  is  an  interpreter  for  a  rule 
language  borrowed  from  CLIPS  (an  idiosyncratic  version  of  LISP),  so  in  general, 
JESS  has  evolved  into  LISP  interpreter  written  in  the  Java  programming  language. 

5.0  LOST  Alternative  Design  Concepts 

Figure  2  provides  a  conceptual  representation  of  the  LOST  problem  domain, 
namely  the  process  (inputs,  decision  and  analysis  activities,  and  outputs)  used  to 
develop  logistics  support  concepts  that  encompass  all  the  ILS  elements.  The  “LOST 
Space”  represents  the  tasks  and  decision  factors  that  a  logistics  manager  must 
consider  as  part  of  a  supportability  analysis  that  is  concerned  with  deriving  “optimal” 
cost-effective  support  concepts,  based  on  the  particular  requirements  of  the  program 
(“LOST  Inputs”). 

As  stated  earlier  in  this  report,  to  our  knowledge  there  are  currently  no 
automated  tools  or  definitive  procedures  (step-by-step  instructions)  that  can 
effectively  help  a  logistics  manager  conduct  a  supportability  analysis  that  1)  starts 
with  system  and  program  requirements,  2)  assesses  these  requirements  against  a 
set  of  “acceptable”  and  applicable  acquisition  logistics  practices  and  procedures,  as 
well  as  legislative,  DoD,  and  service  policies,  etc.,  and  3)  helps  define  the  specific 
tasks  and  decision  factors  that  a  logistics  manager  needs  to  focus  on  to  develop  and 
evaluate  support  concepts  for  their  respective  program. 
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Figure  2.  Conceptual  Representation  of  LOST  Framework 

Based  on  the  research  accomplished  during  the  LOST  requirements  definition 
phase  and  discussed  above,  three  alternative  design  concepts  were  developed  for 
the  LOST.  The  concepts  range  from  simplistic  to  complex  in  terms  of  the  time  and 
level  of  effort  each  would  take  to  fully  design,  develop,  and  test.  The  design  concepts 
considered  for  LOST  included  the  following: 

5.1  Concept  1  -  Support  Analysis  Checklist 

This  checklist  concept  assumes  that  the  decision  process  associated  with  the 
development  of  logistics  support  concepts  for  each  ILS  area  is  “fixed”,  in  essence 
that  the  key  decision  factors  and  tasks  that  comprise  a  supportability  analysis  are 
consistent  across  all  acquisition  programs.  The  LOST  would  output  a  checklist  that 
displays  the  key  decision  factors  and  tasks  that  a  logistics  manager  should  consider 
for  each  ILS  area,  and  support  the  documentation  of  the  specific  status,  rationale  or 
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justification  provided  by  a  logistics  manager  in  response  to  each  decision  factor  or 
task. 

Implementing  the  framework  for  this  design  concept  would  essentially  involve 
two  main  tasks,  namely  1)  developing  the  product/service/source  combinations  (i.e. 
the  support  concepts  for  each  ILS  area,  and  2)  identifying  the  key  tasks  and  decision 
factors  that  are  considered  by  ILS  domain  “experts”  as  part  of  the  process  of 
developing  and  evaluating  logistics  support  concepts. 

This  is  the  most  basic,  and  least  sophisticated  approach  for  designing  and 
developing  the  LOST.  It  does  not  leverage  on  the  implementation  of  any  advanced 
software  technologies,  such  as  those  in  the  artificial  intelligence  domain,  so  the 
technical  risks  are  considered  to  be  low.  It  is  also  probably  the  quickest  concept  to 
implement  in  terms  of  software  design  and  development  time.  This  design  concept 
addresses,  at  a  rudimentary  level,  two  fundamental  requirements  of  LOST,  namely  1) 
implementing  a  structured,  traceable,  and  rigorous  process  for  addressing  key 
decision  factors  and  tasks  that  comprise  a  supportability  analysis,  and  2)  providing 
the  capability  to  document  the  decision  rationale  used  to  address  these  same 
decision  factors  and  tasks.  The  drawback  to  this  concept  is  that  it  does  not  support 
any  automatic,  “smart”  tailoring  of  decision  factors  or  tasks  in  any  of  the  ILS  areas, 
based  on  the  requirements  of  a  specific  acquisition  program. 

5.2  Concept  2  -  LOST  Support  Analysis  Project  Templates 

This  concept  is  an  extension  of  Concept  1  and  assumes  the  decision  process 
associated  with  the  development  of  logistics  support  concepts  for  each  ILS  area  is 
not  “fixed”,  in  essence  that  the  decision  factors  and  tasks  considered  as  part  of  a 
supportability  analysis  are  dependent,  to  some  extent,  on  program  and  system 
requirements.  This  concept  would  be  implemented  through  an  expert-based  system 
design  that  incorporates  “expert  knowledge”  into  the  supportability  analysis  process 
to  help  ensure  program  and  system  requirements  are  addressed  in  a  structured  and 
consistent  manner.  Leveraging  the  knowledge  of  logistics  domain  experts  (e.g. 
Technical  Data),  the  LOST  would  implement  this  knowledge  in  the  form  of  business 
rules  that  would  be  used  to  automatically  generate  the  applicable  task  and  decision 
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factor  templates  that  a  “novice”  logistics  manager  would  need  to  consider  in 
developing  support  concepts  for  their  respective  program.  This  approach  would  help 
define  the  “workspace”  for  logistics  managers  in  order  to  help  them  stay  focused  on 
addressing  only  the  key  tasks  and  decision  factors  that  are  relevant  to  their 
respective  program.  LOST  task  and  decision  factor  templates  would  be  generated 
during  the  initial  setup  or  creation  of  an  LOST  project.  Once  the  project  was  initially 
created,  based  on  “business  rules"  in  the  LOST  knowledge  base,  which  would  be 
used  to  infer  relationships  between  decision  factors  and  tasks,  and  program  and 
system  requirements.  Figure  3  provides  a  summary  representation  of  a  CONOPs  for 
the  LOST  expert  system  concept. 


LogisticsM^^a'jg^r 

►  [  ;  “Creates:  a  Project’? ; 

JniiDSt  - ■■■ 


key  decision  factors  and  activities 


LOST  users  with 
properjirojeciiiccess  privileges  can 
browse  audgenerate  summary 
andiietail  reports  on  the  status  and 
abtionstakeu  to  address  key  project 
decision  factors  and  activities 


Figure  3.  Summary  CONOPs  for  LOST  Design  Concept  #2 

The  inference  engine  would  support  the  task  of  relating  program  and  system 
requirements  to  the  business  rules  in  the  knowledge  base  to  determine  the  pertinent 
ILS  task  and  decision  factor  templates  applicable  to  an  LOST  project.  These 
appropriate  task  and  decision  factor  templates  would  be  attached  to  the  project 
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automatically  by  the  LOST  application  during  the  creation  of  the  project.  Once 
created,  the  logistics  manager  then  opens  the  project  on  the  server  and  works  inside 
these  templates  to  record  or  document  the  status  and  actions  taken  to  address  the 
specific  task  or  decision  factor  identified  by  the  respective  template. 

The  intent  of  this  concept  is  to  try  to  capture  and  model  the  knowledge  of  an 
“expert”  in  terms  of  a  “path”  of  decision  events  that  ties  program  and  system 
requirements  to  specific  logistics  support  analysis  tasks  and  key  decision  factors. 
The  benefit  of  this  concept  is  that  it  gives  deliberate  attention  to  the  more  plausible 
assertion  that  “not  all  acquisition  programs  are  created  equally”.  That  in  fact,  at  least 
some  of  the  key  tasks  and  decision  factors  considered  in  the  process  of  formulating  a 
logistics  support  concept  are  dependent  to  some  extent  on  program  or  system 
requirements  (e.g.  type  of  acquisition  program).  Appendix  3  includes  a  table  of  some 
of  the  more  important  program  and  system  requirements  that  would  be  applicable  to 
the  LOST. 

The  benefit  of  this  concept  is  that  it  also  addresses  two  very  fundamental 
requirements  for  LOST,  namely  1)  implementing  a  structured,  traceable,  and  rigorous 
analysis  process  that  keeps  logistics  managers  focused  on  the  key  tasks  and 
decision  factors  that  impact  their  programs;  and  2)  providing  the  capability  to 
document  the  status  of  tasks  and  decision  rationale. 

One  of  the  primary  drawbacks  to  this  concept  is  that  the  development  of  an 
LOST  knowledge  base,  or  for  that  matter  most  knowledge  bases,  is  that  they  can 
require  a  considerable  amount  of  time  and  resources  to  implement,  since  it  is 
basically  a  knowledge  engineering  process,  and  usually  a  significant  part  of  any 
expert  system  design  and  development  effort.  For  initial  development  and 
demonstration  purposes,  we  would  focus  on  demonstrating  the  capability  for  logistics 
experts  to  build  and  modify  task  and  decision  factor  templates,  and  for  defining  the 
business  rules  that  control  the  assignment  of  these  templates  to  a  specific  LOST 
project  based  on  program  and  system  requirements. 
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5.3  Concept  3  -  Expert-Based,  Support  Concept  Generator 

The  final  concept  is  an  extension  of  the  LOST  expert-based  system  design 
encompassed  as  part  of  Concept  #2.  What  this  approach  adds  to  Concept  #2  is  the 
capability  to  automatically  generate  alternative  support  concepts  for  each  ILS 
element  based  on  program  and  system  requirements  input  by  a  user.  This  could 
prove  to  be  an  extremely  powerful  feature  for  LOST,  since  it  would  go  beyond  just 
generating  the  key  tasks  and  decision  factors  that  a  logistics  manager  needs  to  focus 
on,  it  would  also  provide  the  capability  to  automatically  generate  alternative  support 
concepts  based  on  program  and  system  requirements.  The  benefits  of  this  concept 
include  those  specified  for  Concept  #2,  as  well  as  the  ability  to  extend  the  utility  of 
LOST  to  accommodate  the  analysis  and  generation  of  support  concepts  when 
complete  information  about  program  and  system  requirements  is  not  available, 
primarily  starting  during  the  concept  exploration  phase  of  a  program.  One  of  the 
primary  drawbacks  to  this  concept  is  that  the  development  of  an  LOST  knowledge 
base  becomes  a  more  complex  undertaking,  since  we  are  now  trying  to  develop  the 
capability  for  the  LOST  knowledge  base  and  inference  engine  to  “extrapolate”  from 
what  is  known,  to  accommodate  situations  early  on  in  a  program  when  we  need  to 
develop  and  assess  alternative  support  concepts.  In  this  case,  the  LOST  could  help 
a  novice  logistics  manager  assess  what  alternative  support  concepts  should  be 
considered  and  analyzed  further,  based  on  what  the  user  knows  about  the  program, 
and  eliminate  those  concepts  that  may  not  be  feasible  or  applicable  based  on  current 
program  requirements  and/or  knowledge  acquired  by  “experts”  from  experience  with 
previous  acquisition  programs. 

6.0  Proposed  Solution 

The  three  alternative  design  concepts  discussed  above  were  evaluated 
against  each  key  LOST  functional  requirement  specified  in  Section  2.4.  Based  on 
this  evaluation.  Concept  #2,  “LOST  Support  Analysis  Project  Templates”  was 
determined  to  be  the  most  suitable  concept  for  design  and  development  of  a  LOST 
application.  This  concept  was  embellished  further  to  include  a  definition  of  the  core 


26 


functions  (main  modules)  of  the  LOST,  a  LOST  system  architecture,  and  finally  the 
development  of  a  conceptual,  “walk-through”  demonstration  using  MS  PowerPoint. 

6.1  Main  Application  Modules 

The  LOST  would  be  implemented,  as  a  web-based  application  comprised  of 
the  main  modules  portrayed  in  Figure  4.  LOST  users  would  login  to  the  LOST 
application  and,  depending  on  the  authorizations  specified  in  their  user  profile  (setup 
by  a  system  administrator),  they  would  be  allowed  access  to  one  or  more  of  the 
LOST  modules.  There  are  three  classes  of  users  envisioned  for  the  LOST  including 
a  logistics  manager  that  is  responsible  for  creating  and  updating  task  and  decision 
factor  templates  for  their  respective  LOST  projects;  logistics  domain  “experts”  that 
create  and  maintain  LOST  task,  decision  factor,  and  program  requirement  templates 
and  business  rules  for  applying  the  templates  to  LOST  projects;  and  finally,  “general” 
LOST  users  that  require  only  limited  read/write  capabilities  to  one  or  more  projects  in 
the  LOST  project  database.  This  class  of  “general”  users  would  include  personnel  at 
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1.  Open  a  Project 

2.  Create  a  Project 

3.  User  Admin. 

•  Select  a  Project 

•  Input  Program  and  System 

*  Create  and  Modity  .  , 

•  Select  an  ILS  7\rea 

Requirement  Information 

■  .  ,  ■  ^Jsers  ■ 

*  Update  Status  /  Actions  on 

•  Assign  User  Roies 

:  Decision  Factors  and 

•  Assign  Read/  Write 

Activities  (Tasks) , 

Access  to  LDST 

Projects 

4.  Template  /  Project  Admin. 

5.  RepoHs 

•  Create  and  Maintain  Decision  Factor  and  Activit\' 

•  Generate  System  Admin  Reports 

Templates 

•  Generate  Project  Status  Summary^  and  Detail 

•  Maintain  Program  and  System  Requirement  Input 

Reports  on  Decision  Factors  and  Activities 

Parameters  and  Decision  Logic  for  Q&A 

•  Manage  Project  Artifacts  (Documents,  etc.) 

Figure  4.  LOST  Main  Application  Modules 
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AFSPC,  SMC,  AFMC,  user  locations,  etc.  These  are  stakeholders  or  participants  in 
the  acquisition  process,  but  are  not  directly  assigned  as  logistics  managers  to  a 
specific  program  office,  nor  “experts”  in  any  specific  logistics  domain.  The  primary 
features  of  each  LOST  main  module  are  also  specified  in  Figure  4. 

6.2  LOST  System  Architecture 

Figure  5  depicts  the  overall  system  architecture  envisioned  for  the  LOST 
application.  As  stated  previously,  the  LOST  application  would  be  accessed  through  a 
web-based  browser  utilizing  HTTP  protocols.  Java  Server  Pages  (JSP)  would  be 
“served  up”  from  a  web  server  to  the  user  to  support  the  input  of  data,  template 
display  and  output  reports.  The  LOST  database  would  include  the  templates 
developed  by  “experts”  as  well  as  to  store  LOST  projects  and  project  related 
information  and  documents  (e.g.  ORD).  The  key  feature  of  this  architecture  is  the 


Experts 

Figure  5.  LOST  Expert-Based,  Proposed  System  Architecture 
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separation  of  business  logic  from  the  application  logic  that  would  support  the  creation 
and  modification  of  business  rules  that  control  the  application  of  the  LOST  templates 
to  specific  projects  using  a  product  such  as  the  ILOG  Incorporated,  J-Rules  software 
component.  This  component  allows  “experts”  to  update  business  rules  that  are 
based  on  acquisition  policies  and  directives,  which  could  potentially  change  over 
time.  This  helps  to  reduce  software  sustainment  and  modifications  costs  associated 
with  the  need  for  software  programmers  to  change  “hard-coded”  business  rules  or 
programming  logic  that  are  usually  embedded  as  an  intrinsic  part  of  a  software 
application.  This  is  considered  one  of  the  primary  technical  benefits  of  the  expert- 
based  design  concept  for  the  LOST  application. 

7.0  LOST  Return  on  Investment  Analysis 

A  simplified  Return  on  Investment  (ROI)  analysis  was  undertaken  to  assess 
the  expected  costs  and  benefits  of  LOST.  The  cost  component  of  the  ROI  analysis 
was  comprised  of  estimates  for  the  initial  development  (software  design, 
development,  testing,  and  documentation)  of  the  LOST  application  based  on  the 
features  discussed  in  Section  5  for  Concept  #2  (expert-based  system  utilizing  support 
analysis  templates)  as  well  as  the  projected  sustainment  costs.  The  estimated  total 
cost  of  the  initial  LOST  development  effort  is  $500,000,  with  a  period  of  performance 
of  18  months.  The  annual  maintenance  costs  were  estimated  to  be  $50K  per  year 
with  a  4%  escalation  factor  built-in  each  year  to  account  for  increases  in  labor  costs, 
etc. 

The  benefit  component  of  the  LOST  ROI  analysis  focused  on  developing 
quantitative  cost  savings  for  two  key  benefit  sub-components  (or  categories)  - 
reduced  acquisition  logistics  training  costs,  and  reduced  time  associated  with 
conducting  logistics  support  analysis  activities  in  a  program  office.  Table  1  provides 
a  summary  of  the  specific  cost  factors  used  to  derive  an  annualized  cost  savings  at 
SMC  for  the  LOST.  The  values  for  “Estimated  Time  Reduction”  were  provided  by 
SMC/AXLX  personnel  based  on  current  and  historical  experience  with  acquisition 
training  and  the  time  expended  by  logistics  managers  in  program  offices  performing 
acquisition  logistics  support  activities.  For  simplicity,  military  personnel  costs  were 
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based  on  salary  factors  provided  in  AFI  65-103  (FY  2001  rates)  for  a  Captain  (0-3), 
and  GS-13  Civilian. 


Estimated 

Current 

Estimated 

Estimated 

Time 

Savings 

Savings 

Time 

Time 

Number 

Savings 

(Capt) 

(GS-13) 

Annual 

Category 

Reduction 

Months 

Personnel 

(Months) 

Mil 

Civ 

Total 

Training 

30% 

12 

10 

36 

$257,097 

0 

$257,097 

Acquisition 

Logistics 

Support 

Total 

25% 

12 

0 

3 

$42,850 

$88,525 

$131,375 

$388,472 

Table  1.  Parameters  Used  to  Derive  LOST  Annual  Cost  Savings 

The  LOST  cost  savings  for  “Training”  are  based  on  an  estimated  30% 
reduction  in  the  time  devoted  to  initial  acquisition  logistics  courses  required  to  obtain 
Level  II,  Acquisition  Logistics  Certification,  which  on  the  average,  takes  about  one 
year  to  complete.  Based  on  guidance  from  SMC/AXLX,  it  was  estimated  that  12 
military  (no  civilian)  personnel  are  involved  in  Acquisition  Logistics  training  in  a  typical 
year.  Using  the  30%  training  time  reduction  factor,  and  other  parameters  cited  in 
Table  1  for  “Training”,  the  baseline  annual  cost  savings  is  approximately  $257,000 
per  year. 

The  LOST  cost  savings  for  “Acquisition  Logistics  Support”  in  Table  1  are 
based  on  an  estimated  25%  reduction  in  the  time  devoted  by  SMC  logistics 
managers  to  analyzing  and  documenting  logistics  support  requirements  for  their 
respective  programs.  This  also  includes  the  time  expended  by  SMC/AXLX  staff 
“experts”  who  provide  specialized  support  in  one  or  more  of  the  ILS  functional 
domains  (e.g.  Technical  Data,  Maintenance  Planning,  Training  Systems,  etc.)  to 
logistics  managers  in  one  or  more  SMC  program  offices.  SMC/AXLX  estimated  that 
through  the  use  of  the  LOST,  logistics  managers  could  reduce  the  overall  total  time 
expended  to  the  analysis  and  documentation  of  support  requirements  (during  EMD) 
by  25%  or  from  one  year  to  nine  months.  SMC/AXLX  also  projected  that  it  had  an 
average  of  six  programs  (including  major  modifications)  in  the  EMD  phase  at  any 
given  time.  For  simplicity,  it  was  assumed  that  four  of  the  six  programs  ongoing  at 


30 


any  given  time  at  SMC  are  supported  by  four  civilian  (GS-13),  and  two  military 
(Captain)  logistics  managers.  Using  these  parameters,  as  well  as  the  salary  factors 
cited  earlier,  the  estimated,  annual  cost  savings  (baseline)  for  LOST  is  approximately 
$388,500.  An  annual  3%  cost  of  living  increase  was  applied  to  this  figure  to  project 
out-year  costs. 

Figure  6  portrays  a  ten-year  cost-benefit  analysis  that  compares  the  annual 
estimates  for  LOST  software  development  and  maintenance  to  projected  LOST  cost 
savings  estimates.  Based  on  starting  LOST  development  in  FY02,  the  projected 
“break-even”  point  would  occur  towards  the  end  of  FY05:  at  which  point  the  initial 
development  costs  for  LOST  would  be  recouped. 


LOST  Cost  /  Benefit  Analysis 


Fiscal  Year 


-------  Cost  — ■ —  Benefit  ?  Payback 


Figure  6.  LOST  Cost-Benefit  Analysis  Summary 


8.0  Conclusion 

The  LOST  research  focused  on  defining  and  developing  a  framework  for  a 
decision  support  tool  that  can  help  acquisition  logistics  managers  perform  a  more 
stmctured,  focused,  and  rigorous  supportability  analysis  for  space  systems  programs. 
The  proposed  LOST  design  concept  will  address  this  need  by  leveraging  advances  in 
both  web-based  and  expert  system  software  technologies,  as  well  as  the  limited  pool 
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of  logistics  domain  expertise,  to  provide  a  decision  support  application  that 
emphasizes  the  active,  versus  passive,  management  of  “expert”  knowledge  that  can 
help  novice  logistics  managers  focus  on  addressing  critical  support  analysis  tasks 
and  decision  factors  that  can  significantly  impact  the  sustainment  of  space  systems. 
The  LOST  design  concept  proposed  in  this  report  will: 

■  Improve  the  process  for  conducting  logistics  support  analysis  early  on  and 
throughout  the  life-cycle  program  by  providing  a  structured  approach  for  addressing 
and  documenting  the  logistics  support  analysis  decisions  and  tasks. 

■  Ensure  that  logistics  support  analysis  tasks  and  decision  factors  are 
addressed  in  a  more  structured  and  consistent  manner,  thereby  reducing  the 
workload  and  training  requirements  for  logistics  managers. 

■  Address  the  need  to  capture,  and  build-on  critical  corporate  knowledge 
from  logistics  “experts”  who  will  be  leaving  the  workforce  in  the  near  future  by 
providing  a  mechanism  for  capturing  and  retaining  corporate  knowledge  embodied  by 
a  limited  pool  of  acquisition  logistics  “experts”  in  a  readily  maintainable  format. 

It  is  worth  noting  that  during  the  LOST  research  effort,  HQ  AFSPC/LGXR 
representatives  identified  the  need  to  be  able  to  conduct  a  low-level  of  fidelity, 
sustainability  and  affordability  analysis  for  programs  early  on  in  a  program.  This  type 
of  analysis  would  focus  on  expanding  the  functionality  of  the  proposed  LOST  design 
concept  outlined  in  this  report  to  help  staff  personnel  at  HQ  AFSPC  develop  an  initial 
set  of  logistics  requirements  that  would  be  documented  in  an  ORD,  and  precede  the 
analysis  of  logistics  support  requirements  at  SMC.  The  intent  would  be  to  determine 
cost,  readiness,  and  performance  impacts  of  supportability  decisions  early-on  in  a 
new  program  (or  major  modification)  before  these  same  decisions  are  translated  to 
program  requirements  that  subsequently  get  specified  as  part  of  an  ORD  or  other 
program  requirements  document.  In  this  case,  the  LDST  could  be  expanded  to 
include  a  structured,  consistent,  rigorous  logistics  requirements  definition  process 
that  could  be  used  by  HQ  AFSPC  personnel  and  ensure  the  right  requirements  are 
addressed  in  the  acquisition  process,  particularly  in  the  specification  of  contract 
deliverables  that  have  a  direct  impact  on  the  long  term  sustainment  costs  and 
performance  of  space  systems. 
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Appendix  1 .  LOST  Key  Points  of  Contact 


Name 

Organization 

E-Mail 

DSN 

Commercial 

*  Mr.  Jeff  Wampler 

AFRL/HESS 

Jeff,  warn  oler(®woafb.  af .  m  i  I 

785-7773 

937-255-7773 

Mr.  Ed  Boyle 

AFRL/HESS 

Edward.bovle®woafb.af.mii 

785-5169 

937-255-5169 

Mr,  Greg  Adams 

SMC/AXL 

Systems 

Acquisition 

Carl.adamsflilosanaeles.af.  mil 

310-363-1974 

Mr.  John  Cox 

SMC/  AXLX 

833-5433 

310-363-5433 

Ms.  Grade  Wantland 

MILSATCOM 

Tech  Data 

Gracie.wantland(®losanaeles.af.mil 

833-6424 

310-363-6424 

Lt  Col  Paul  Scholte 

AFSPC/LGXR 

Paul.scholteOoeterson.af.mil 

692-3861 

719-554-3861 

Mr.  Dale  McKinzie 

AFSPC/LGXR 

Dale.  mckinzieOoeterson.af .  mil 

692-3923 

719-554-3923 

Ms.  Sue  Glass 

SMC/AXLY 

Training  and 
Training  Support 

sue.alass@Josanqeles.af.mil 

833-5463 

310-363-5463 

Capt  Janet  Haug 

SMC/AXLY 

Training  and 
Training  Support 

Janet.haua@losanaeles.af,mil 

833-3603 

310-363-3603 

Capt  Tim  Fromm 

SMC  Det11 
(DAG) 

Tim.fromm@cisf.af.mil 

834-2051 

719-556-2051 

Mr.  Mike  Newman 

SMC/AXLM 

Maintenance 

Planning 

Michael.newman@losanaeles.af.mil 

833-0290 

310-363-0290 

*  Mr.  Pat  Vincent 

TASC  Inc 

Pivincent@tasc.com 

NA 

937-426-1040 

x489 

937-426-1040 

x447 

Mr.  Terry  Jenkins 

Bd  Systems 

Tienkins@tor.bdsvs.com 

NA 

310-618-8798 

*  Project  Manager 
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Appendix  2.  LOST  Questionnaire 
Requirements  Definition 


Problem  Definition  and  Scope 

Please  address  each  of  the  following  statements  and  questions  regarding  the  LOST.  The 
statements  and  questions  in  this  first  iteration  are  intended  to  help  reach  a  consensus  among 
the  members  of  the  LOST  focus  group  on  the  following: 

■  Formulating  a  clear,  concise  problem  definition  for  the  LOST. 

■  Scoping  the  focus  of  the  problem  to  aid  in  defining  the  functional  and  technical 
requirements  for  the  LOST. 

■  Identifying  contributing  factors  to  the  LOST  problem  that  will  serve  to  help  better 
define  LOST  goals,  guide  the  remainder  of  work  in  the  requirements  definition 
process  (Note:  Some  of  these  factors  could  be  translated  directly  into  goals  or 
requirements  for  the  LOST). 

REMARKS”  ARE  STRONGLY  ENCOURAGED  FOR  ALL  QUESTIONS/STATEMENTS 

■  LOST  Problem  Definition 

Space  Systems  program  offices  cannot  adequately  justify  and  defend  logistics 
support  decisions  made  during  the  acquisition  process  to  show  that  the  decisions  made 
provide  for  the  delivery  of  a  supportable,  sustainable  system  which  achieves  the 
performance,  cost,  and  schedule  goals  of  the  program. 

Note:  “Logistics  support  decisions”  as  identified  in  the  problem  statement  above 
refers  to  the  concept  or  strategy  (organic,  contractor,  mix  of  organic  and  contractor,  TSPR) 
proposed  by  the  SPO  for  supporting  both  on  and  off-equipment  maintenance  requirements  at 
the  organizational  and/or  depot  level. 

Agree  Disagree  Neither  Agree/Disagree 

Remarks: 

■  Assumptions 

1 .  In  order  to  expedite  the  development  of  design  concepts  for  LOST,  the  initial  focus  of 
the  LOST  will  be  on  supporting  the  development  of  logistics  support  concepts  or  strategies 
derived  during  the  system  acquisition  process,  starting  with  a  Milestone  0  decision  to  enter 
the  Concept  Exploration  phase,  and  ending  with  a  Milestone  III  decision  to  enter  Production, 
Fielding/Deployment,  Operational  Support. 

Agree  Disagree  Neither  Agree/Disagree 

Remarks: 

2.  In  order  to  expedite  the  development  of  design  concepts  for  LDST,  the  initial  focus  of 
an  LDST  requirements  analysis  should  address  the  development  of  logistics  support 
concepts  or  strategies  for  the  qround  segment  of  a  space  system. 

Agree  Disagree  Neither  Agree/Disagree 
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Remarks: 


3.  The  primary  users  of  the  LOST  would  be  Acquisition  Logistics  Managers  (LMs)  in 
space  system  program  offices  (SPOs),  or  other  Acquisition  Logistics  staff  personnel 
responsible  for  developing  logistics  support  concepts  or  strategies. 

Agree  Disagree  Neither  Agree/Disagree 

Remarks: 

4.  A  baseline  process  for  developing  a  logistics  support  concept  or  strategy  that 
addresses  key  decision  criteria/issues,  includes  source  data  including  inputs  (e.g.  program 
requirements,  legislative  requirements,  policy  and  directives,  and  other  decision  factors), 
activities  (e.g.  the  questions  or  issues  that  must  be  addressed);  and  outputs  (products  of  an 
activity),  can  be  clearly  defined. 

Agree  Disagree  Neither  Agree/Disagree 

Remarks: 

5.  The  final  output  of  the  LDST  process  is  the  recommended  support  concept  or 
strategy  for  a  space  system  segment? 

Agree  Disagree  Neither  Agree/Disagree 

Remarks: 

■  LDST  Problem  Definition  -  Contributing  Factors 

The  following  set  of  statements  is  an  attempt  to  identify  contributing  factors  to  the  LDST 
problem  (as  defined  above)  that  prevent  a  SPO  from  adequately  justifying  and  defending 
logistics  support  decisions.  These  factors  (once  embellished  or  expanded)  will  help  form  the 
baseline  for  defining  specific  goals  for  the  LDST. 

1 .  No  feasible  capability  exists  to  clearly  show  or  trace  the  specific  steps  or  activities 
accomplished  during  the  logistics  support  planning  process. 

Agree  Disagree  Neither  Agree/Disagree 

Remarks: 

2.  No  feasible  capability  exists  to  document  the  specific  rationale  used  at  each  key 
decision  point  in  the  logistics  support  planning  process. 

Agree  Disagree  Neither  Agree/Disagree 

Remarks: 

3.  No  feasible  capability  exists  to  delineate  all  the  decision  factors  in  a  program  that 
should  be  considered  (by  policy,  directive,  etc.)  in  the  logistics  support  planning  process. 

Agree  Disagree  Neither  Agree/Disagree 
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Remarks: 


4.  No  feasible  capability  to  bring  all  stakeholders  (to  include  users,  AFSPC,  sustainment 
logistics  personnel,  contractors,  etc.)  into  the  process  early-on  in  a  program  to  collaborate 
and  provide  inputs  to  decisions  made  in  the  logistics  support  planning  process. 

Agree  Disagree  Neither  Agree/Disagree 

Remarks: 

5.  The  frequency  of  personnel  turnover  (due  to  retirements,  force  reductions,  PCS 
actions,  etc.)  into  and  out  of  LM  or  acquisition  logistic  positions  responsible  for  logistics 
support  planning  significantly  impacts  (negative)  the  process. 

Agree  Disagree  Neither  Agree/Disagree 

Remarks: 

6.  The  skill  level  or  experience  level  of  LMs  or  acquisition  logistic  personnel  responsible 
for  logistics  support  planning  can  significantly  impact  the  decisions  made  during  the  logistics 
support  planning  process. 

Agree  Disagree  Neither  Agree/Disagree 

Remarks: 

■  General  Questions 

1.  In  developing  a  logistics  support  concept  or  strategy,  does  the  decision  process  (in 

terms  of  activities,  inputs,  outputs)  differ  by  the  type  of  space  system  under  study  or 
development?  Yes  No 

Remarks: 

2.  In  developing  a  logistics  support  concept  or  strategy;  does  the  decision  process  (in 

terms  of  activities,  inputs,  outputs)  differ  by  the  type  of  space  system  segment  (ground, 
launch,  or  space)  under  study  or  development?  Yes  No 

Remarks: 

3.  Assuming  we  can  define  a  baseline  process  for  the  logistics  support  planning 

process,  can  we  further  define  a  discrete  set  of  business  rules  that  could  be  used  to  help 
tailor  the  baseline  process  for  “unique”  acquisition  programs?  Yes  No 

Remarks: 

4.  What  do  you  feel  are  some  of  the  key  benefits  of  an  LDST  that  can  address  the 
factors  identified  above  (e.g.  reduce  the  acquisition  cycle  time,  improve  continuity  in  the 
logistics  support  planning  process,  reduce  training  time  for  LM  s,  reduce  life  cycle  costs, 
etc.)? 

5.  Should  the  LDST  target  “expert"  or  “novice”  Acquisition  LMs,  or  both? 
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6.  In  your  opinion,  will  it  be  possible  to  define  the  characteristics  of  an  “expert”  and 

“novice"  LM  or  Acquisition  Logistic  user  for  LOST?  Yes  No 

Remarks: 

7.  In  your  opinion,  are  ALL  the  relevant  data  /  information  sources  available  for 

developing  inputs  (decision  criteria  /  factors,  etc.)  for  a  LOST?  Yes  No 

Remarks: 

8.  Identify  some  of  the  more  relevant  references  (e.g.  legislative  directives,  policies,  etc.) 
a  LM  needs  to  consider  in  developing  a  logistics  support  strategy. 

9.  What,  if  any,  relevant  data  /  information  sources  do  you  feel  are  not  readily  available 
for  developing  inputs  (decision  criteria  /  factors,  etc.)  for  a  LOST? 
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Appendix  3.  LOST  Data  Requirements 


The  data  elements  identified  below  represent  a  subset  of  the  program  and  system 
data  elements  that  would  be  used  from  program  documents  such  as  the  Operational 
Requirements  Document  (ORD),  Mission  Need  Statement  (MNS),  Program  Management 
Directive  (PMD),  Acquisition  Strategy  Plan  (ASP),  etc.  to  create  a  LDST  project.  Logistics 
domain  “experts”  would  use  these  data  elements  to  construct  decision  (business)  rules  that 
would  be  applied  in  LDST  to  control  the  assignment  of  task  and  decision  factor  templates  to 
LDST  projects. 


Data  Element 

Sources 

Remarks 

Program  Title 

ORD  Title 

Type  Acquisition 

PMD/ASP 

Type  System 

ORD 

Number  of  Systems 

PMD 

Type  Missions 

ORD,  Section  1 

Maintenance  Concept 

ORD,  Section  1 

Levels  of  Maintenance 

ORD,  Section  5a 

Organizational,  Intermediate,  Depot 

Type  Maintenance  Tasks 

ORD,  Section  5a 

Remove  and  Replace,  etc. 

Expected  Operational  Life 

ORD 

Provisioning  Strategy 

ORD,  Section  5f 

Special  PHS&T 
Considerations 

ORD,  Section  5f 

Support  Equipment 

ORD,  Section  5b 

System  Description 

ORD,  Section  1 

ORD,  Section  5e 

ORD,  Section  5g 

Mission  Capable  Rate  - 
Objective 

ORD,  Section  4c 

MC  Rate  (Objective)  -  Peacetime 

MC  Rate  (Objective)  -  Wartime 

Mission  Capable  Rate  - 
Threshold 

ORD,  Section  4c 

MC  Rate  O’hreshold)  -  Peacetime 

MC  Rate  (Threshold)  -  Wartime 

MTBM  -  Scheduled  - 
Objective 

ORD,  Section  4c 

MTBM  Scheduled  (Objective)  -  Peacetime 
MTBM  Scheduled  (Objective)  -  Wartime 

MTBM  -  Scheduled  - 
Threshold 

ORD,  Section  4c 

MTBM  Scheduled  (Threshold)  -  Peacetime 
MTBM  Scheduled  (Threshold)  -  Wartime 

MTTR  -  Scheduled 
Objective 

ORD,  Section  4c 

MTTR  Scheduled  (Objective)  -  Peacetime 
MTTR  Scheduled  (Objective)  -  Wartime 

MTTR  -  Scheduled 
Threshold 

ORD,  Section  4c 

MTTR  Scheduled  (Threshold)  -  Peacetime 
MTTR  Scheduled  (Threshold)  -  Wartime 

Operational  Availability 
Objective 

ORD,  Section  4c 

Ao  Peacetime  (Objective) 

Ao  Wartime  (Objective) 

Operational  Availability  - 
Threshold 

ORD,  Section  4c 

Ao  Threshold  -  Peacetime 

Ao  Threshold  -  Wartime 

Human  Factors  and 
Performance 

Requirements 

ORD,  Section  5e 

Manpower  Constraints 

ORD,  Section  5e 

Missions 

ORD,  Section  1 
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Data  Element 

Sources 

Remarks 

Unique  Data 

Requirements 

ORD,  Section  5f 

Unique  Infrastructure  and 
Environmental 

Compliance  Requirements 

ORD,  Section  5f 

Logistics  Support 
Constraints 

MNS,  Section  5 

Note  1:  ORD  reference  is  CJCSI  3170.01A,  Appendix  A,  Enclosure  E,  10  Aug  1999 

Note  2:  MNS  reference  is  AP1 10-602,  Attachment  5 

Note  3:  PMD  reference  is  HQ  USAF  Operating  Instruction  800-2 
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